Partial Order Knowledge Structures for CAT Applications

Bayesian and graph models of student knowledge assessment have made significant progress in the last decadeand are challenging the more traditional IRT approach for CAT applications. We review some of the most prominent frameworks in Bayesian knowledge assessment and how they compare to IRT and introduce one such framework in the family of Bayesian models, the POKS (Partial Order Knowlege Structure). A comparison of the POKS approach to IRT and a Bayesian Network approach showed that it can perform detailed knowledge assessment at a computational cost of orders of magnitude less than a Bayesian Network and IRT. The assessment accuracy results of experiments show that it is at least as good as a one-dimensional IRT model and generally outperforms a Bayesian Network with small data sets. However, a number of challenges remain for the POKS approach as well as for other Bayesian frameworks in CAT applications. One of the most important issue is how scalable the approaches are over a large number of items. Another issue is the estimation of reliability and error margins, which are currently almost ignored by these approaches. We review these challenges and the work ahead. Acknowledgment Presentation of this paper at the 2007 Conference on Computerized Adaptive Testing was supported in part with funds from GMAC®. Copyright © 2007 by the Authors All rights reserved. Permission is granted for non-commercial use. Citation Desmarais, M. C., Pu, X, & Blais, J.-G. (2007). Partial Order Knowledge Structures for CAT Applications. In D. J. Weiss (Ed.), Proceedings of the 2007 GMAC Conference on Computerized Adaptive Testing. Retrieved [date] from www.psych.umn.edu/psylabs/CATCentral/ Author Contact Michel C. Desmarais or Xiaoming Pu, Polytechnique Montreal, C.P. 6079, succ. Centre-Ville, Montréal, Québec, Canada, H3C 3A7. Email : [email protected], [email protected], [email protected] Partial Order Knowledge Structures for CAT Applications Adaptive testing can be considered one of the first applications of what is currently a very active research topic, that of adaptive and personalized interfaces. The principle of adaptive interfaces is based on constructing a user model of an application and then adapting the performance of the application to the model. This is exactly what adaptive testing does. It constructs a model of the respondent’s knowledge and adapts the administered items as a function of the model, generally with the specific goal of making a knowledge assessment with a minimum number of items. Although the area of adaptive interfaces includes a wide range of adaptation, from user preferences to user intentions (McTear 1993), it remains that computerized adaptive testing (CAT) might be the very first large-scale application of the basic principle of these applications: to construct a user model and adapt the performance of the application as a function of the model. The domains of CAT and adaptive interfaces evolved separately from each other, largely unaware of each other’s developments. The area of adaptive interfaces and, in particular, that of adaptive learning environments gave rise to several models of learner knowledge and several techniques for its evaluation (Self, 1988). While item response theory (IRT) was rapidly developing in the area of adaptive testing, the area of intelligent tutorials was developing its own approaches for representing and assessing competences for adaptive learning environments (Carr & Goldstein, 1977). Most of these efforts were based on rule-based systems and aimed at providing a very detailed assessment of learner knowledge. The main feature of these models was to arrive at an accurate assessment that referred, not only to the precise concepts mastered, but also to incorrect concepts, or mal-rules (Payne & Squibb, 1990). These models had the advantage of providing a high level of granularity in that they could provide a very precise assessment of acquired or missing knowledge/competences; however, they did not integrate any notion of uncertainty, which is inherent to the modeling of knowledge. Inversely, work in the area of psychometrics and IRT models incorporated, from the outset, the notion of uncertainty and were devoted mainly to estimating the reliability of the models and the confidence intervals used to make an assessment with a known degree of certainty. However, the granularity of IRT-based models still remains low and generally limited to one dimension, or, in the case of more recent work on multidimensional IRT, to a few dimensions simultaneously, well below the level of granularity that can be attained with the rule-based models of intelligent tutorial environments. These divergences are easily explained considering that, in the case of psychometrics, the most frequent requirements originate from the context of summative evaluation and consist in determining whether the respondent will pass or fail a test. The requirements of intelligent tutorial environments are aimed, instead, at determining the learning problems of the learner in order to select very specific capsules of pedagogical content aimed at remedying incorrect concepts or guiding the learner toward a more advanced content. The respective requirements of the two domains are, therefore, very different, which explains in large part the little influence they have had on each other. The link between the two domains emerged from work on graphical models of knowledge and we mention, among others, that of Almond and Mislevy (1999), Mislevy and Gitomer